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ABSTRACT:  Training  programs  for  warfighters  have  become  increasingly  dependent  on  the  use  of  simulation 
systems,  made  necessary  by  the  growing  costs  of  live  flight  training.  The  transition  to  a  greater  dependence  on 
simulators  has  required  an  increase  in  concerns  about  the  ability  of  simulators  to  provide  effective  training.  Research 
in  the  area  of  simulator  fidelity  has  focused  primarily  on  training  capabilities,  evaluating  current  systems  for  strengths 
and  weaknesses.  The  evaluation  of  current  systems  is  an  essential  component  in  providing  the  best  possible  training 
environment  to  pilots;  however,  the  impact  of  improvements  is  of  equal  importance.  An  informed  decision  about  which 
changes  will  have  the  greatest  benefit  for  warfighter  readiness  would  provide  a  means  of  maximizing  the  return  on  the 
investment  in  the  development  of  the  system  and  creating  the  most  efficient  and  effective  training  environment  possible. 
The  current  work  presents  a  process  by  which  system  deficiencies  and  potential  solutions  are  identified  and  then 
integrated  into  quantitative  models  based  on  multiple  factors.  The  factors  included  in  the  model  are  potential  for 
improvement,  training  gaps,  extent  to  which  solutions  address  deficiencies,  material  costs  of  solutions,  and  time  costs  of 
solutions.  A  project  manager  and  engineers  provided  feedback  in  three  areas:  the  extent  to  which  a  solution  addresses 
a  deficiency,  the  cost  of  the  technology,  and  the  engineering  hours  involved  in  the  change.  Three  models  were  computed 
and  compared  across  solutions,  revealing  the  utility  of  the  empirical  process.  The  models  included  in  the  comparison 
varied  the  weights  within  and  across  factors,  establishing  the  importance  of  determining  appropriate  weightings.  The 
process  and  algorithm  presented  in  the  current  work  have  far  reaching  practical  applications  as  a  tool  for  assisting  in 
the  decision-making  process  during  development  and  improvement  of  training  technologies.  Ultimately,  a  cost-benefit 
decision-making  process  will  provide  an  effective  tool  for  maximizing  training  benefits  during  the  improvement  of 
simulation  systems. 


1.  Introduction 

The  use  of  simulators  to  supplement  live-flight  training 
has  been  established  as  an  effective  and  cost  efficient 
method  for  increasing  warfighter  readiness  [1,2,3],  Aging 


aircraft  and  budget  constraints  have  increased  the 
dependence  on  simulation  systems  as  a  training  method. 
The  growing  demand  for  simulation  systems  to  provide 
suitable  training  across  the  diversity  of  skills  required  for 
combat  readiness  has  encouraged  technological  advances 
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in  simulators.  The  advances  include  improvements  to  the 
training  experience  within  a  given  simulation  system  and 
advances  in  networking  systems  together  to  increase  the 
scope  of  the  training  environment  [4].  Rapid  changes  in 
available  technology  permit  increasingly  sophisticated 
system  development  and  improvement.  Although 
simulation  training  is  designed  to  address  very  specific 
training  demands  for  pilot  readiness  and  the  need  for 
effective  training  is  great,  the  method  of  improvement  of 
our  simulation  systems  often  lacks  evidence-based 
decision-making  [5], 

In  previous  research,  the  authors  presented  a  method  of 
incorporating  research  into  models  to  inform  decision¬ 
making  [5],  The  goal  of  the  method  is  to  provide  an 
empirical  process  to  inform  decision-making  regarding 
improvements  to  simulation  systems.  The  process  can  be 
structured  to  maximize  training  benefits,  while 
maintaining  an  accounting  of  system  and  budget 
constraints.  The  method  leveraged  work  from  a 
competency-based  approach  to  training,  in  which  the 
skills,  knowledge,  and  experiences  required  for  pilots  to 
become  combat  mission  ready  are  identified  [6].  The 
experiences  from  the  Mission  Essential  Competencies 
(MEC)  approach  were  used  to  evaluate  the  current 
training  capabilities  and  deficiencies  of  a  Deployable 
Tactical  Trainer  (DTT)  under  development  at  the  Air 
Force  Research  Laboratory  (AFRL).  Study  findings 
revealed  that  the  incorporation  of  multiple  factors  into 
decision-making  models  affected  the  relative  importance 
of  identified  deficiencies. 

Prost,  Schreiber,  and  Bennett  [5]  compared  three  models 
to  investigate  the  utility  of  quantifying  research  data  for 
the  decision-making  process.  The  models  investigated 
included  a  frequency  model,  an  improvement  model,  and 
an  improvement  model  that  accounted  for  training  gaps. 
The  frequency  model  computed  the  number  of  times  that 
a  deficiency  was  identified  as  the  primary  detriment  to 
training  across  all  MEC  experiences.  The  model  allowed 
for  a  direct  comparison  of  deficiencies  by  comparing  the 
proportion  of  endorsements  for  each  deficiency  relative  to 
the  total  number  of  responses.  The  improvement  model 
weighted  the  frequencies  by  the  degree  to  which  each 
MEC  experience  could  be  improved  for  the  simulation 
system.  Finally,  the  improvement  model  that  accounted 
for  training  gaps  weighted  the  frequencies  by  the 
improvement  score  and  an  additional  weighting  based  on 
whether  or  not  the  experience  represented  a  training  gap. 
Results  of  the  study  showed  that  a  comparison  across 
models  revealed  differential  importance  of  the 
deficiencies,  depending  on  whether  or  not  factors  were 
included  in  the  models.  The  identification  of  system 
deficiencies  and  the  quantification  of  the  impact  of  the 
deficiencies  on  the  training  capabilities  of  the  system 
provided  an  important  step  in  building  a  data-driven 
process  to  assist  decision-making.  The  final  step  in  the 


development  of  the  process  is  to  extend  the  work  from  the 
identification  and  modeling  of  the  deficiencies  to  address 
potential  solutions. 

2.  Current  Work 

The  current  study  presents  a  process  for  identifying, 
evaluating,  and  comparing  deficiencies  of  a  simulation 
system,  and  consequently  the  solutions  for  addressing 
those  deficiencies.  The  purpose  of  the  process  is  to 
provide  project  managers,  researchers,  and  engineers  with 
a  tool  to  quantify  the  factors  that  influence  the 
improvement  process.  Ideally,  simulator  system 
improvements  should  maximize  increases  in  training 
capabilities  while  considering  factors  such  as  current 
training  gaps,  the  degree  to  which  a  solution  addresses  a 
deficiency  in  the  system,  cost  of  improvements,  and 
integration  of  the  solutions  into  the  current  system. 

The  work  presented  below  specifically  focuses  on 
integrating  1)  the  extent  to  which  a  solution  addresses  a 
deficiency  and  2)  the  costs  involved  in  the  solution  into 
previous  models  that  evaluated  room  for  improvement 
and  training  gaps.  Costs  taken  into  consideration  include 
material  costs  and  time  costs.  To  examine  the  effects  of 
the  two  factors  of  interest  in  this  study  on  the  empirically 
driven  decision-making  process,  data  from  previous 
studies  on  room  for  improvement  and  training  gaps  will 
be  included  in  the  current  study  for  the  air-to-air  skill 
area.  The  air-to-air  skill  area  will  be  examined  in  the 
current  study  for  presentation  purposes  only,  although  the 
process  would  extend  to  the  entire  set  of  skills  in  a 
practical  application. 

The  evidence-based  decision-making  process  is  examined 
by  comparing  multiple  models  to  determine  the  effect  of 
adding  factors  for  consideration.  The  models  can  be 
divided  into  two  classes:  Deficiency  Models  and  Solution 
Models.  Deficiency  Models  come  from  previous  work 
comparing  factors  important  in  determining  the  impact  of 
identified  deficiencies  [5],  Solution  Models  were 
developed  for  the  current  study  as  a  means  of  producing 
quantifiable  scores  for  each  solution.  The  scores  for  each 
model  represent  the  magnitude  of  training  improvement 
that  would  be  realized  by  the  integration  of  the  solution 
for  one  model,  and  then  the  improvement  relative  to  the 
costs  for  the  other  two  models.  The  purpose  of  the 
presentation  of  the  Solution  Models  in  the  current  study  is 
to  illustrate  the  ability  to  quantify  the  effect  of 
improvements,  relative  to  costs,  in  such  a  way  that  the 
solutions  can  be  directly  compared.  The  capacity  of  the 
proposed  process  to  quantify  the  factors  involved  in  the 
decision-making  process  can  remove  some  of  the 
guesswork  from  the  development  and  improvement  of 
simulation  systems. 
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3.  Method 

The  current  study  integrates  engineer  survey  information 
about  technological  changes  to  a  deployable  trainer  into  a 
modeling  framework  for  evaluating  the  effect  of 
deficiencies  on  a  system.  Due  to  the  integration  of  new 
factors  into  an  existing  model,  some  data  presented  below 
is  from  previously  published  work  [5]. 

3.1  Evaluators 

Ratings  from  previous  research  by  seven  F-16  subject 
matter  experts  (SMEs)  were  used  in  the  current  study.  The 
SMEs  completed  two  surveys.  One  was  an  evaluation  of 
the  fidelity  of  the  DTT,  and  the  other  was  an  evaluation  of 
the  deficiencies  of  the  DTT.  In  addition  to  the  data  from 
previous  work,  three  evaluators  participated  in  the  current 
study  by  providing  responses  to  a  survey  designed  to 
assess  aspects  of  integrating  new  technology  to  resolve 
deficiencies  of  the  DTT.  All  evaluators  were  provided 
with  detailed  documentation  of  system  components, 
capabilities,  and  limitations,  and  were  provided  multiple 
opportunities  to  fly  various  missions  in  the  system  before 
providing  system  ratings. 

The  seven  evaluators  from  the  previous  research  were  all 
male,  all  recently  retired  (3  years  on  average)  F-16  pilots 
from  the  United  States  Air  Force  (USAF),  and  had  an 
average  number  of  operational  F-16  flight  hours  of  2,1 19 
(average  total  flight  hours  of  3,495)  [5], 

The  project  manager  overseeing  the  development  of  the 
DTT  completed  the  survey  for  the  current  study.  The 
solutions  provided  were  identified  and  discussed  with  two 
additional  engineers  working  on  the  development  of  the 
DTT.  All  participating  individuals  have  been  involved  in 
the  development  of  the  DTT  since  the  first  working 
system. 

3.2  Surveys 

Data  were  used  from  a  total  of  three  surveys.  Two  of  the 
surveys,  the  Fidelity  and  Deficiency  surveys,  were 
administered  during  previous  research.  The  final 
Solutions  survey  was  developed  specifically  to  expand  the 
previous  model.  The  two  surveys  administered  in  the 
context  of  previous  research  are  briefly  described  below 
(and  can  be  seen  in  more  detail  in  Reference  5).  The 
survey  used  in  the  current  research  is  described  in  greater 
detail  below. 

Responses  to  two  surveys  (previously  described  as 
Fidelity  and  Deficiency  surveys)  were  used.  The  surveys 
were  constructed  by  leveraging  MEC  research  for  the  F- 
16.  The  original  surveys  used  197  items:  70  air-to-ground 
(A/G)  MEC  experiences,  55  suppression  of  enemy  air 
defense  (SEAD)  MEC  experiences,  45  air-to-air  (A/A) 
MEC  experiences,  and  27  bold  face  EPs  for  the  F-16. 
The  current  work  focuses  specifically  on  the  A/A  skill 


area  of  the  system;  therefore,  the  fidelity  survey  data  from 
the  45  items  in  that  skill  area  were  included  in  the  current 
data  set. 

The  Fidelity  Survey  consisted  of  SMEs  providing  each  of 
the  items  on  the  survey  with  a  rating  from  0  through  5,  on 
the  following  scale: 

0=  N/A  (Capability  to  experience  does  not  exist) 

1=  Capability  to  experience  exists,  but  is  very  poor 
2=  Capability  to  experience  exists,  but  is  poor 
3=  Capability  to  experience  exists,  but  is  marginal 
4=  Capability  to  experience  exists,  and  is  good 
5=  Capability  to  experience  exists,  and  is  very  good 

The  Deficiency  Survey  asked  SMEs  to  identify  the 
deficiency  with  the  greatest  influence  on  each  of  the  197 
survey  items.  Respondents  could  choose  from  a  list  of  1 7 
deficiencies  that  were  previously  identified.  For  the 
purposes  of  this  study,  because  the  focus  was  on  the  air- 
to-air  skill  area,  the  top  five  deficiencies  for  A/A  were 
included.  The  most  influential  deficiencies  for  the  A/A 
skill  area  were: 

1)  Limited  out-the-window  (OTW)  capability 

2)  No  helmet  mounted  sight  (HMS)  capability 

3)  No  EP  instructor/operator  station 

4)  No  electronic  countermeasure  (ECM) 
capability 

5)  Software  issue,  no  weapon  system  capability 

The  Solutions  survey  was  designed  to  investigate  the 
potential  solutions  for  each  of  the  deficiencies.  Surveys 
solicited  identification  of  the  three  most  promising 
solutions  available  for  each  of  the  top  five  deficiencies  in 
the  A/A  skill  area.  Respondents  then  evaluated  a  series  of 
qualities  for  each  of  the  solutions  identified.  The 
following  were  the  questions  presented  for  each  solution: 

1)  To  what  extent  does  the  proposed  solution 
remedy  the  deficiency? 

2)  What  is  the  projected  cost  in  dollars  for  the 
solution? 

3)  What  is  the  projected  cost  in  time  for  the 
solution? 

The  Solutions  survey  attempted  to  identify  the  utility  and 
cost  of  each  particular  solution.  To  understand  the  cost  of 
each  solution,  it  was  important  to  further  differentiate 
costs  along  the  dimensions  of  material  and  time. 

3.3  Apparatus:  Current  DTT  system  (taken  from  [5]) 

The  DTT  system  (see  Figure  1)  that  was  evaluated 
consisted  of  the  following  major  hardware/software 
functionalities: 

Hardware:  F-16  cockpit  shell  with  three  out-the-window 
30-inch  displays,  the  actual  F-16  stick/throttle,  and 
simulation  of  all  cockpit  displays  and  switch  functions  on 
a  high  resolution  23”  interactive  touch  screen  display. 
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Image  generator  was  an  SDS  International  AAcuity®  PC- 
IG  system.  Brief/debrief  included  SmartBoard  and  two 
50-inch  displays  for  Head-Up  Display,  Radar  Warning 
Receiver,  and  Multi-Function  Display. 


Figure  1.  Example  of  DTT  simulation  system 


Software:  The  system  uses  classified  Block  30  (SCU  5p) 
actual  operational  F-16  software.  One  database  is 
currently  installed  and  available.  Debrief  software  has  the 
ability  to  link  and  time-synchronize  video  recordings 
from  multiple  players.  It  also  has  the  ability  to  network 
through  Distributed  Interactive  Simulation  (DIS)  or  High 
Level  Architecture  (HLA)  standards.  It  has  chaff/flare 
capability,  but  no  ECM.  Some  classified  weapons  systems 
are  available,  such  as  Joint  Direct  Attack  Munition 
(JDAM). 

3.4  Procedure 

The  Solutions  Survey  was  given  to  the  project  manager 
for  the  DTT  project.  The  project  manager  was  asked  to 
collaborate  with  appropriate  engineers  to  complete  the 
survey.  Survey  responses  were  entered  into  a  spreadsheet 
for  analysis.  Previous  data  was  combined  with  the 
Solutions  Survey  data  to  compute  multiple  models  for 
comparison.  Solution  Models  were  computed  using  data 
from  the  Improvement  and  Training  Gap  Deficiency 
Model  and  responses  from  the  Solutions  Survey. 
Computation  of  models  was  based  on  including  one 
additional  factor  for  each  model. 

There  were  a  total  of  six  factors  included  in  the  models 
presented.  The  first  three  factors  (Deficiency  Factors) 
were  from  previous  research,  but  necessary  to  compute 
the  current  models  (taken  from  Reference  5).  The  factors 
were  defined  as  follows: 

Deficiency  Factors: 

•  Deficiency  Evaluation  was  based  on  the 
frequency  of  response  for  each  deficiency  by  the 
evaluators  for  each  skill. 

•  Fidelity  Evaluation  was  based  on  the  proportion 
of  improvement  possible  for  each  skill. 


calculated  as  five  minus  the  average  fidelity 
rating  for  the  skill  divided  by  five. 

•  Training  Gap  Evaluation  was  based  on  MEC 
research  [6]  establishing  the  training  gaps  for  the 
skills.  The  weightings  were  1  for  no  gap,  2  for 
potential  gap,  and  3  for  gap. 

Solution  Factors: 

•  Extent  of  Solution  was  based  on  a  survey 
response  from  0  to  5  (Not  at  All  to  Completely). 

•  Dollar  Cost  was  based  on  a  survey  response  from 
0  to  5  (Prohibitive  to  Minimal). 

•  Time  Cost  was  based  on  a  survey  response  from 
0  to  5  (Prohibitive  to  Minimal). 

Using  the  data  available  from  previous  survey  collection 
and  the  data  collected  in  this  study,  a  series  of  models  was 
computed  and  then  compared.  A  total  of  six  models  were 
computed.  Of  the  six  models,  three  were  Deficiency 
Models  and  three  were  Solutions  Models. 

Model  1:  Frequency  Model.  In  Model  1,  the  proportions 
presented  represent  the  total  frequency  of  responses  for  a 
deficiency  across  all  items,  divided  by  the  total  number  of 
responses  (description  from  [5]). 

Model  2:  Improvement  Model.  In  Model  2,  the 
proportions  presented  are  the  sum  of  the  weighted  scores 
for  each  deficiency  divided  by  the  total  score  across  all 
deficiencies  and  items.  The  weighted  scores  in  Model  2 
are  computed  by  multiplying  the  frequency  of  a 
deficiency  response  by  the  proportion  for  improvement 
potential  (description  from  [5]). 

Model  3:  Improvement  and  Training  Gap  Model.  In 
Model  3,  the  proportions  presented  are  the  sum  of  the 
weighted  scores  for  each  deficiency  divided  by  the  total 
score  across  all  deficiencies  and  items.  These  weighted 
scores  were  computed  by  first  multiplying  the  frequency 
of  a  deficiency  response  by  the  proportion  for 
improvement  potential,  and  then  multiplying  that  product 
by  a  weight  identifying  the  level  of  training  gap  present 
for  each  skill  (description  from  [5]). 

Model  4:  Extent  of  Deficiency  Resolution  Model.  In 
Model  4,  solutions  are  presented  for  each  of  the 
deficiencies.  A  score  is  given  to  each  solution  based  on 
the  Model  3  score  representing  the  impact  of  the 
deficiency  on  the  system.  Each  solution  score  for  Model  4 
uses  the  corresponding  Model  3  deficiency  score, 
weighted  by  the  extent  to  which  the  solution  resolves  the 
deficiency.  The  extent  of  the  resolution  is  calculated  as 
the  proportion  of  the  maximum  resolution,  based  on  the 
ratings  provided  (i.e.,  the  rating  divided  by  5). 

Model  5:  Material  Cost  of  Solution  Model.  In  Model  5, 
the  score  for  each  solution  is  calculated  as  the  Model  4 
score  weighted  by  the  dollar  cost  rating  for  the  solution. 
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The  weighting  for  dollar  cost  was  calculated  as  a 
proportion  of  the  minimal  cost,  based  on  the  rating  for 
dollar  cost  (i.e.,  rating  divided  by  5).  The  resulting  score 
represents  the  amount  of  improvement  relative  to 
monetary  costs. 

Model  6:  Time  Cost  of  Solution  Model.  In  Model  6, 
solution  scores  were  calculated  using  Model  5  score 
weighted  by  the  time  cost  rating  for  the  solution.  The 
weighting  for  time  cost  was  calculated  as  a  proportion  of 
the  minimal  cost,  based  on  the  rating  for  time  cost  (i.e., 
rating  divided  by  5).  The  resulting  score  represents  the 
amount  of  improvement  relative  to  monetary  and  time 
costs,  allowing  for  comparison  across  solutions. 

4.  Results 

Analyses  comparing  the  relative  importance  of  the 
deficiencies  for  each  of  the  Deficiency  Models  were 
conducted  previously  [5],  Table  lpresents  the  proportion 
of  the  total  deficiency  score  for  each  model  that  could  be 
attributed  to  the  specific  deficiencies. 


Table  1.  Proportion  of  total  score  attributable  to  each 
deficiency  for  air-to-air  experiences  for  each  of  the  three 
Deficiency  Models 


Model  1 

Model  2 

Model  3 

Limited  OTW 

0.54 

0.47 

0.47 

NoHMS 

0.10 

0.09 

0.09 

No  EP 

0.09 

0.13 

0.09 

No  ECM 

0.06 

0.08 

0.07 

No  Weaps 

0.05 

0.08 

0.15 

The  first  part  of  the  Solution  Survey  task  on  the  air-to-air 
skill  area  of  the  F-16  simulation  system  was  to  identify 
potential  solutions  for  each  of  the  deficiencies.  The 
project  manager  and  engineers  identified  up  to  three 
solutions  for  each  deficiency.  The  following  solutions 
were  those  presented  and  then  evaluated  by  participants. 

Limited  out-the-window  field  of  vision: 

•  Procure  and  integrate  a  mini-DART; 

•  Procure  and  integrate  a  DART  system; 

•  Partial  Dome. 

No  Helmet  Mounted  Sight  Capabilities: 

•  Procure  and  implement  a  currently  available 
solution  with  mini-DART; 

•  Procure  and  implement  a  currently  available 
solution  with  DART; 

•  Partner  with  AF  developed  system  utilizing 
actual  operational  HMCS  helmets  and 
commercial  video  cards. 

No  emergency  procedure  instructor/operator  station: 

•  Integrate  existing  MTT  IOS; 

•  Develop/Write  IOS. 

No  electronic  countermeasure  capabilities: 


•  Propose  and  make  software  changes  and 
program  enhancements  to  cockpit  code  to 
support  and  implement  physics  based  ECM  such 
as  an  external  ECM  pod,  decoys,  EMI  on  FCR, 
etc.; 

•  Rather  than  using  physics  based  modeling, 
implement  simulation  and  emulation. 

Missing  weapons  in  the  database: 

•  Integrate  available  code  to  support  desired 
weapons; 

•  Develop  and  write  code  to  support  a  specific 
weapon  capability. 

The  next  step  in  the  analyses  of  the  Solutions  Models  was 
to  take  survey  ratings  and  transform  them  for 
incorporation  into  models.  Table  2  presents  the  ratings 
provided  for  each  solution  to  Solutions  Survey  questions. 
The  ratings  represent  the  value  reached  by  consensus  for 
the  project  manager  and  engineers  participating  in  the 
evaluation. 


Table  2.  Ratings  for  Solutions  Survey  questions  across 
proposed  solutions. 


Survey  Ratings 

Extent 

Dollar 

Time 

Limited 

OTW 

Mini-DART 

3 

1 

4 

DART 

5 

0 

3 

Partial  Dome 

2 

2 

1 

Available  with 
mini-DART 

3 

1 

5 

NoHMS 

Available  with 
DART 

5 

1 

5 

Partner  with 

AF  and 

commercial 

cards 

5 

3 

3 

No  EP 

Use  existing 

IOS 

5 

5 

4 

Develop  IOS 

5 

4 

1 

No  ECM 

Implement 
physics  based 
software 

4 

4 

0 

Implement 
simulation  and 

emulation 

2 

3 

1 

No 

Weaps 

Integrate 
available  code 

5 

5 

4 

Develop  code 

5 

5 

1 

The  Solutions  Survey  scores  were  transformed  and 
incorporated  into  the  corresponding  models.  Three 
models  were  computed  using  the  Deficiency  Model  3 
scores  and  the  weights  from  the  Solutions  Survey  ratings. 
Table  3  presents  the  model  scores  for  each  solution  across 
the  three  models  computed.  The  model  scores  represent 
the  potential  improvement  to  the  system  for  a  solution. 
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The  large  change  in  model  scores,  from  model  4  to 
models  5  and  6,  is  the  introduction  of  costs.  The 
deficiency  of  limited  out-the-window  visuals,  although 
having  the  greatest  impact  on  the  system,  is  also  the  most 
cost  intensive  deficiency  to  address. 


Table  3.  Model  scores  for  each  solution  across  the  three 
models  evaluated. 


Model  Scores 

Model 

Model 

Mode] 

4 

5 

6 

Limited 

OTW 

Mini-DART 

0.28 

0.06 

0.04 

DART 

0.47 

0.00 

0.00 

Partial  Dome 

0.19 

0.07 

0.01 

Available  with 
mini-DART 

0.05 

0.01 

0.01 

No  HMS 

Available  with 
DART 

0.09 

0.02 

0.02 

Partner  with 

AF  and 

commercial 

cards 

0.09 

0.05 

0.03 

No  EP 

Use  existing 

IOS 

0.09 

0.09 

0.07 

Develop  IOS 

0.09 

0.07 

0.01 

NoECM 

Implement 
physics  based 
software 

0.05 

0.04 

0.00 

Implement 
simulation  and 
emulation 

0.03 

0.02 

0.00 

No 

Weaps 

Integrate 
available  code 

0.15 

0.15 

0.12 

Develop  code 

0.15 

0.15 

0.03 

5.  Summary 

The  results  of  the  current  study  build  on  previous  research 
to  provide  a  more  comprehensive  model  for  evaluating 
the  influence  of  simulation  improvements.  Previous 
findings  showed  the  importance  of  considering  how  much 
improvement  could  be  gained  for  target  training 
experiences  and  whether  or  not  training  experiences  were 
in  areas  with  current  training  gaps.  The  results  from  the 
previous  study  found  that  the  relative  importance  of  the 
deficiencies  of  a  system  varied  when  considering  those 
two  factors.  The  current  study  incorporates  additional 
important  factors  to  the  previous  model:  the  extent  to 
which  a  solution  addresses  a  deficiency  and  the  cost  of  the 
solution  [5]. 

A  comparison  of  the  models  revealed  that  a  data  driven 
decision-making  process  is  an  effective  means  of 
differentiating  among  solutions.  Examining  changing 
scores  for  solutions  across  models,  there  is  a  dramatic 
shift  in  the  training  benefits  to  the  system  for  solutions. 
For  example,  if  materials  and  time  costs  were  not  an 


issue,  any  solution  for  the  limited  OTW  field  of  vision 
deficiency  would  address  more  of  the  overall  training 
deficiencies  of  the  system  for  air-to-air  experiences  than 
would  solutions  for  other  deficiencies.  Once  materials  and 
time  costs  are  taken  into  consideration,  solutions  to  other 
deficiencies  are  revealed  as  having  a  greater  effect  on  the 
training  environment,  suggesting  that  those  solutions  are 
more  economical.  The  example  presented  in  this  study 
illustrates  the  capacity  of  this  process  to  provide  data 
driven  models  for  comparison;  however,  an  important 
point  of  this  example  is  to  remember  that  the  weightings 
used  in  the  current  study  between  and  within  factors  were 
arbitrarily  set  to  compare  models.  For  example,  the 
weighting  given  to  materials  costs  was  the  same  as  the 
weighting 

To  increase  the  accuracy  of  the  models  produced  through 
this  process,  two  changes  would  be  implemented  when 
the  process  is  used  in  a  real  world  environment.  First, 
rather  than  using  ratings  to  address  questions  of  cost,  true 
value  estimates  would  be  used.  The  dollar  and  hour  values 
would  transform  those  factors  into  continuous  variables, 
rather  than  discrete  ratings.  Continuous  values  for  those 
factors  would  provide  for  dramatically  greater  accuracy  in 
comparison  across  solutions.  The  second  change  would  be 
to  transform  cost  estimates  relative  to  the  overall  budget. 
Because  budget  constraints  represent  the  framework  for 
development,  the  proportion  of  the  total  budget  that  a 
solution  requires  provides  a  more  useful  scale  for 
comparison  of  the  effect  of  solutions.  These  changes 
would  increase  the  internal  accuracy  of  the  model; 
however,  equally  important  is  the  accuracy  of  the  model 
in  regard  to  including  all  relevant  factors. 

The  evidence-based  decision-making  process  presented  in 
the  current  manuscript  incorporates  multiple  sources  of 
data  into  a  single  data  driven  model.  The  final  models 
examined  included  information  about  the  current  fidelity 
of  a  simulation  system,  the  existing  training  gaps  of  a 
system,  the  extent  to  which  a  solution  resolves 
deficiencies,  and  costs  of  a  solution.  For  presentation 
purposes,  two  costs  were  presented:  material  and  time 
costs.  A  comprehensive  model  would  also  include 
additional  solution  costs,  such  as  maintenance  costs, 
engineering  risk  costs,  compatibility  with  current 
simulator  technologies,  and  pilot  training  costs.  These 
additional  costs,  which  may  also  influence  the 
effectiveness  and  efficiency  of  training,  can  be 
incorporated  into  models  just  like  the  two  types  of  costs 
included  in  the  current  study.  Ultimately,  multiple  data 
driven  models  could  be  compared  to  determine  the  best 
simulator  improvements,  depending  on  a  wide  range  of 
criteria. 

The  work  presented  in  the  current  manuscript  explores 
one  step  in  a  process  for  evaluating  and  improving 
simulation  systems.  The  growing  dependence  on 
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simulation  systems  has  introduced  many  questions  about 
the  capabilities  of  simulation  training  versus  live  flight 
training,  as  well  as  situations  in  which  simulation  training 
is  effective  [8,  9,  10].  The  ability  to  evaluate  and  improve 
simulation  systems  through  a  precise  empirical  process 
will  begin  to  answer  questions  about  how  to  appropriately 
supplement  training  via  simulators.  Additionally, 
simulation  system  improvements  can  be  undertaken  more 
precisely  in  order  to  meet  the  rapidly  changing  demands 
on  the  systems. 
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